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1.  Introduction 

DUCS  is  an  experimental  multi-level  architecture  that  provides  a  distributed  representation  for 
frame-like  concept  structures.  (The  name  stands  for  Dynamically  Updatable  Conceptual  Structures.) 
The  ultimate  goal  of  this  research  is  to  develop  a  powerful  short  term  memory  that  can  construct  and 
manipulate  concepts  rapidly.  Such  a  memory  could  play  an  important  role  in  neural  network  model;  of 
higher  level  cognitive  phenomena  such  as  language  understanding  or  problem  solving.  As  an  example, 
consider  how  one  might  represent  the  knowledge  that  “Down  by  the  hen  house,  John  threw  a  rock  at  the 
fox.”  With  a  case  frame  representation1,  this  sentence  could  be  represented  by  the  following  structure: 


AGENT 

JOHN 

VERB 

THROW 

OBJECT 

ROCK 

DESTINATION 

FOX 

LOCATION 

HEN  HOUSE 

In  this  paper  I  show  how  such  structures  can  be  realised  in  DUCS  as  distributed  activity  patterns 
over  a  space  of  neuron-like  computing  elements.  DUCS  can  retrieve  a  slot  filler  from  its  concept  buffer 
given  the  slot  name;  it  can  also  retrieve  entire  concepts  from  its  concept  memory  given  some  name/filler 
pairs  as  cues.  In  order  to  be  useful  as  a  short  term  concept  memory,  DUCS  was  designed  to  have  the 
following  properties: 

•  Concepts  can  be  created,  modified,  or  deleted  simply  by  changing  an  activity  pattern;  it  is  not 
necessary  to  go  through  a  series  of  incremental  weight  changes  and  resettlings. 

•  Concepts  can  contain  any  number  of  slots.  Accuracy  of  retrieval  degrades  gradually  as  the  number 
of  slots  stored  exceeds  the  capacity  of  the  network. 

•  Individual  slots  of  a  concept  can  be  accessed  by  associative  retrieval.  If  the  pattern  for  a  slot  name 
doesn't  exactly  match  what  was  previcusly  stored,  e.g.,  if  we  try  to  access  an  elephant’s  TRUNK  slot 
by  referring  to  his  NOSE,  assuming  the  two  slot  names  have  similar  but  not  identical  microfeatures 
the  retrieval  will  succeed  and  simultaneously  the  query  will  be  corrected  to  read  TRUNK. 

•  More  than  one  slot  of  a  concept  can  be  accessed  simultaneously,  e.g.,  it  is  po?:ible  to  access  the 
AGENT  and  the  OBJECT  slots  of  an  action  in  parallel. 

•  Concepts  can  be  associatively  retrieved  by  combining  several  name/filler  pairs  into  one  cue.  For 
example,  if  John  did  things  in  various  places  and  if  several  things  happened  at  the  hen  house,  to 

‘McClelland  and  Kawamoto  (1986)  have  alio  explored  connectionist  case  frame  representations.  Their  model  performs 
lexical  disambiguation  and  case  role  assignment  for  single  sentences  using  four  Axed  cases. 
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recall  what  action  John  took  at  the  hen  house  one  combines  the  cues  AGENT:  JOHN  and  LOCATION: 
HEN  HOUSE. 

•  Concepts  that  have  been  accessed  recently  remain  in  the  concept  memory,  while  those  that  have 
not  been  accessed  fade  out. 

Perhaps  the  simplest  way  to  implement  concept  structures  (not  the  way  chosen  by  DUCS)  is  as  a 
collection  of  three-part  bit  vectors  of  form  (concept-tag  slot-name  slot-filler).  The  tags  could  be 
meaningful  patterns  or  randomly-generated  ones;  their  purpose  is  to  group  together  slots  belonging  to 
the  same  concept.  A  collection  of  these  tripartite  patterns  could  be  stored  in  an  associative  memory,  and 
given  a  concept  tag  and  slot  name  it  would  be  easy  to  retrieve  the  corresponding  filler.  However,  this 
approach  lacks  several  other  properties  important  for  a  short  term  concept  memory. 

One  issue  is  the  need  to  access  multiple  slots  in  parallel.  In  a  conventional  associative  memory,  such 
as  Hopfield’s  or  Willshaw’s  models  (Hopfield,  1982;  Willshaw,  1981),  information  resides  in  the  weights 
between  units.  In  order  to  access  several  slots  in  parallel  one  must  make  several  copies  of  the  a.«or<^ive 
network  and  keep  the  weights  identical  in  each  copy.  An  alternative  is  to  represent  concepts  as  activity 
patterns  rather  than  as  weights,  with  the  activity  pattern  transmitted  from  a  central  buffer  to  a  number 
of  input/output  groups  for  retrieving  different  components  simultaneously.  DUCS  uses  this  technique. 

A  second  issue  is  the  ability  to  associatively  retrieve  a  particular  concept  by  combining  multiple 
name/filler  pairs  to  form  a  cue.  This  also  appears  to  require  an  activity-based  representation  for  slots. 
One  might  attempt  to  concatenate  all  the  slots  of  a  concept  to  form  one  huge  vector  in  order  to  effect 
this  sort  of  retrieval,  but  since  concepts  contain  a  varying  and  unbounded  number  of  slots,  and  the  space 
of  possible  slot  names  is  quite  large,  that  approach  would  not  be  viable. 

Finally,  since  the  available  biological  evidence  is  that  activity  patterns  can  be  modified  more  quickly 
than  weights,  it  makes  sense  to  explore  activity-based  representations  when  designing  a  short  term  working 
memory.  The  solution  presented  here  is  one  of  the  key  features  of  DUCS:  it  has  error  correcting  and 
associative  retrieval  properties  similar  to  a  classical  associative  memory,  but  it  is  organised  on  different 
principles. 

2.  Overview  of  the  Architecture 

At  the  lowest  level  of  DUCS,  individual  slots  are  created  by  setting  up  an  activity  pattern  in  one  of 
several  slot  name  groups  and  its  associated  slot  filler  group;  see  Figure  1.  The  name  and  filler  patterns 
then  jointly  cause  a  pattern  to  appear  in  the  selector  group.  Selector  groups  function  as  a  combination 
pullout  network  (Moser,  1984;  Touretsky,  1986;  Touretsky  Sc  Hinton,  1986)  and  viewpoint  mapping 
device  (Hinton,  1981a);  they  do  the  real '  *  A  storing  and  retrieving  things.  The  detailed  structure  of 

the  various  groups  will  be  described  in  the  -»*  ection. 

Slots  are  collected  into  concepts  in  DUCS  by  superimposing  in  the  concept  buffer  the  activity  patterns 
of  all  the  selector  groups.  Conversely,  slots  are  extracted  from  the  concept  buffer  by  clamping  their  names 
into  slot  name  groups  and  allowing  the  selector  and  slot  filler  groups  to  settle;  the  corresponding  filler 
for  each  slot  then  appears  in  the  associated  slot  filler  group. 

The  highest  level  of  DUCS  is  the  concept  memory,  which  can  hold  multiple  concepts  simultaneously. 
New  concepts  enter  the  memory  one  at  a  time  through  the  concept  buffer.  Concepts  can  be  recalled  by 
combining  some  slots  in  the  concept  buffer  and  using  the  resulting  activity  pattern  as  a  cue  to  associatively 
retrieve  and  fill  in  the  rest  of  the  concept  buffer  pattern  from  concept  memory. 
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Figure  1:  The  DUCS  Architecture. 


3.  A  Glimpse  of  the  Details 

This  section  presents  a  concise  summary  of  the  wiring  pattern  of  DUCS.  To  begin,  slot  names  and  slot 
fillers  are  N- bit  and  F-bit  binary  feature  vectors,  respectively,  appended  to  their  logical  complements. 
That  is,  a  slot  name  5  is  a  2W-bit  vector  where  a <  =  0(i+Ar)  for  1  <  i  <  N,  and  a  slot  filler  v  is  a  2F-bit 
vector  where  =  V(i+f)  for  1  <  t  <  F.  The  selector  groups  and  the  concept  buffer  are  both  organized 
as  4 F  x  2r  arrays  of  units,  where  R  is  a  parameter  between  0  and  N.  The  arrays  are  4 F  bits  wide 
because  slot  fillers  are  stored  redundantly  —  each  bit  and  its  complement  are  represented  twice.  Simple 
voting  circuitry  shown  in  Figure  2  is  used  to  arrive  at  the  correct  slot  filler  vector  during  retrieval  even 
when  some  of  the  concept  buffer  or  selector  units  are  in  error.  Such  errors  are  unavoidable  when  using 
distributed  representations,3  as  DUCS  does,  unless  memory  is  kept  very  sparse,  because  slots  will  overlap 
in  the  concept  buffer.  The  error  rate  for  retrieval  of  individual  slots  is  a  function  of  the  number  of  slots 
superimposed  in  the  concept  buffer,  and  also  of  the  "breadth*  of  the  buffer,  which  is  a  function  of  R. 
For  N  —  20  and  F  =  20,  with  R  =  4  the  concept  buffer  has  a  breadth  of  16  units  and  can  store  three 
to  four  slots  before  retrieval  errors  occur.  Increasing  R  makes  the  representation  more  sparse,  thereby 

reducing  the  chance  for  slots  to  overlap  and  cause  errors.  The  accuracy  of  the  selector  groups  can  also  _ 

be  increased  by  widening  the  concept  buffer  to  6 F  or  8 F  units.  While  the  degree  of  sparseness  will  be  r 
unchanged,  the  extra  redundancy  aids  error  correction. 


J  Hinton,  McClelland,  and  Rumelhart  (1988)  describe  some  properties  and  applications  of  distributed  representations. 


Storing  filler  v  in  the  slot  named  a  generates  a  4F  x  2R  pattern  over  the  selector  units  {s,,,}.  Each 
bit  Vi  is  copied  into  one  of  the  2R  locations  in  column  t  of  the  selector  array,  and,  independently,  into  one 
of  the  2r  locations  in  column  t  +  2 F.  A  unit’s  position  j  within  a  column  is  based  on  a  randomly-chosen  _ 
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Figure  2:  Error  correction  circuitry  to  exploit  the  redundant  storage  of  slot  fillers.  The  figure  shows  bit 
Vi  of  a  slot  filler,  its  complement  bit  v^,  and  the  corresponding  selector  units  for  four  copies  of  the  bit 
(two  of  Vi  and  two  of  .) 

/2-bit  subset  of  a;  a  different  subset  is  chosen  for  each  of  the  4 F  columns.  All  selector  groups  use  the 
same  subsets  for  their  corresponding  columns.  Figure  3  shows  the  wiring  for  mapping  a  single  slot  filler 
bit  Vi  into  some  Si.7  and  from  there  to  the  corresponding  bit  in  the  concept  buffer.  Here  R  =  2  so 
the  the  value  of  j  is  determined  by  two  slot  name  bits,  ax  and  av.  Note  that  v,  will  also  be  copied  into 
column  »  +  2F  in  some  position  Ac,  where  k  is  determined  by  a  (probably)  different  pair  of  slot  name  bits 
at  and  au.  Also,  which  is  v<  will  be  copied  into  some  position  in  columns  i  +  F  and  *  +  3 F,  based 

on  two  other  pairs  of  slot  name  bits.  This  redundancy  is  necessary  not  only  for  error  correction,  but  also 
for  associative  retrieval,  as  will  be  explained  shortly. 

Selector  units  function  as  a  skeleton  filter.3  The  units  in  each  column  t  are  organized  in  a  mutually 
inhibitory  winner-take-all  network  such  that  the  winner  will  be  the  unit  that  receives  excitation  on  every 
one  of  its  R  inputs  from  the  slot  name  group.  The  winning  Sj,7  functions  as  a  mapping  unit  allowing 
activation  to  pass  between  the  concept  buffer  unit  6j,;  and  the  slot  filler  unit  v;. 

DUCS  uses  non-linear  units  with  variable  gain  and  outputs  restricted  to  the  unit  interval;  all  connec¬ 
tions  are  symmetric.  This  is  commonly  known  as  a  Hopfield  and  Tank  model  (Hopfield  &  Tank,  1985). 
To  retrieve  a  slot,  the  name  is  loaded  into  the  slot  name  group  and  then  it  and  the  concept  buffer  are 
clamped,  with  the  gain  initially  set  very  low.  As  the  net  settles,  the  selector  units  will  copy  the  states  of 
the  chosen  A,./s  (1  <  i  <  4 F)  to  the  slot  filler  group’s  v/s.  The  variable  gain  feature  is  important  for 
getting  the  retrieval  to  work  properly,  especially  when  the  slot  name  does  not  exactly  match  what  was 
stored,  but  the  gain  can  rise  quickly:  fewer  than  20  cycles  are  required  for  a  retrieval 

Although  increasing  R  increases  the  capacity  of  the  concept  buffer,  smaller  R  values  make  it  easier  to 
retrieve  a  slot  baaed  on  an  inexact  name  match.  For  example,  suppose  we  try  to  retrieve  the  contents  of 
Clyde  the  elephant’s  TRUNK  slot  using  the  pattern  for  NOSE  instead.  If  the  two  patterns  differ  in  a  single 
microfeature,  then  of  the  (‘^)  distinct  A-bit  subsets  of  the  TRUNK  pattern,  the  fraction  that  differ  from 
NOSE  is  R/N.  Assuming  R  <  N,  this  fraction  is  small.  Let  slot  filler  unit  v,  be  one  that  happens  to  be 
mapped  using  the  bit  where  the  two  patterns  differ.  Due  to  this  difference,  some  selector  a;,*  rather  than 

3So  called  because  only  a  skeleton  subset  of  the  units  are  enabled  at  any  one  time.  Hinton  (1981b)  describes  the  use  of 
skeleton  Altera  in  a  connectionist  implementation  of  a  semantic  network.  Sejnowski  (1981)  presents  arjuments  for  the 
existence  of  skeleton  Altera  in  the  brain. 
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Figure  3:  Hashing  one  copy  of  the  slot  filler  bit  into  one  of  2R  concept  memory  bits  based  on  a  random 
/2-bit  subset  of  the  slot  name.  In  the  figure,  R  =  2.  All  connections  have  positive  weights  unless  shown 
otherwise.  Mutually  inhibitory  connections  among  the  2R  selector  units  have  been  omitted  for  clarity. 

Sj.y  will  most  likely  be  the  winner  in  column  »,  and  so  u,  may  be  pressured  to  take  on  the  value  of  fc,.* 
rather  than  bi,j.  But  if  R  <.  N,  chances  are  very  good  that  the  redundant  copies  of  Vi  represented  by 
selectors  in  columns  *  +  F,  i  +  2 F,  and  »  +  3 F  probably  chose  subsets  that  didn’t  include  the  erroneous 
bit,  and  so  these  three  selectors  will  be  mapped  correctly  and  apply  pressure  on  v,  to  assume  the  correct 
value.  In  turn,  v<  will  try  to  make  s,.y  rather  than  s<,k  the  winner  in  column  i,  and  due  to  the  symmetry 
of  connections,  this  will  apply  pressure  to  change  the  state  of  the  erroneous  slot  name  bit.  When  the  slot 
name  group  is  unclamped  part  way  through  the  process  of  raising  the  gain  during  retrieval,  the  pattern 
will  spontaneously  change  from  NOSE  to  TRUNK.  In  preliminary  experiments  using  N  =  30,  F  =  100, 
and  R  =  5,  the  slot  name  did  in  fact  correct  itself  during  retrieval,  even  with  two  or  three  bit  errors. 

4.  Design  of  the  Weights 

AH  weights  and  thresholds  in  DUCS  are  fixed.  Their  signs  are  predetermined  by  the  model,  and  their 
magnitudes  are  estimated  automatically  when  the  network  is  initialized,  based  on  the  values  of  F,  N, 
and  R.  A  slight  amount  of  empirical  tuning  is  still  required  due  to  the  simplicity  of  the  current  weight 
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estimation  algorithm,  which  does  not  take  into  account  the  fact  that  the  number  of  connections  a  slot 
name  unit  makes  with  selector  units  varies  with  F.  (The  expected  value  is  2F  2r  R/N,  but  there  is 
some  variance  because  the  4 F  columns  of  the  concept  buffer  and  selector  group  choose  their  R- set3  of 
slot  name  units  randomly.)  Another  complicating  factor  is  the  fact  that  the  amount  of  input  each  slot 
name  and  slot  filler  unit  receives  depends  on  the  number  of  slots  that  make  up  the  concept,  which  varies 
from  one  concept  to  another  but  is  generally  expected  to  increase  with  R. 

Slot  retrieval  is  a  constraint  satisfaction  problem.  The  formula  for  choosing  weights  is  designed  to 
balance  the  constraints  that  affect  each  type  of  unit.  For  example,  selector  unit  Sj.y  should  be  affected 
equally  by  concept  buffer  unit  6,.j,  slot  filler  units  t>,  and  Vi+F,  and  the  ensemble  of  R  slot  name  units 
to  which  the  selector  unit  connects.  The  selector  must  also  be  responsive  to  inhibitory  input  from  the 
other  2r  -  1  selectors  in  column  t.  A  table  of  weights  and  their  signs  is  given  below.  All  connections  are 
bidirectional,  but  the  number  of  connections  from  each  unit  of  type  x  to  units  of  type  y  is  not  necessarily 
the  same  as  the  number  from  y  to  x. 


#  from  #  from 


Connection  Type 

Symbol 

Sign 

x  to  y 

y  to  x 

slot  filler  «-*  complement 

Vi  ~  Vi+p 

- 

1 

1 

slot  name  —  complement 

a,  *-*  Hi+N 

- 

1 

1 

selector  *—  concept  buffer 

3i.j  ^i.j 

+ 

1 

1 

selector  <-»  slot  name 

3i.j  ■*“'  <3* 

+ 

R 

~2F2r 

selector  *-*  slot  filler 

S..J  —  Vi,  Vi  +  tF 

+ 

2 

2  x  2r 

seL  —  slot  filler  compl. 

3i.]  *-*  Vt  +  p,  Vi  +  3 F 

- 

2 

2x2r 

seL  *-*  competing  sel. 

»i.i  «-  ».,*  U  *  k ) 

- 

2r  -  1 

2r  -  1 

5.  Organization  of  the  Concept  Memory 

The  concept  memory  holds  a  collection  of  concepts,  which  are  patterns  of  length  4 F  times  2R  obtained 
by  viewing  the  two-dimensional  concept  buffer  as  a  one-dimensional  vector.  To  retrieve  a  concept,  one  or 
more  cues  are  loaded  into  slot  name/slot  filler  groups  and,  through  the  selector  groups,  superimposed  in 
the  concept  buffer.  The  concept  buffer  then  has  a  partial  pattern  which  can  serve  as  a  retrieval  cue  for 
the  entire  concept.  In  the  present  version  of  DUCS  the  concept  memory  is  a  Willshaw-style  associative 
memory  (Willshaw,  1981),  except  that  the  one-bit  weights  of  Willshaw’s  model  have  been  replaced  with 
simple  processing  units.  These  units  can  be  used  to  detect  failed  retrievals  (when  no  stored  concept 
matches  the  name/ filler  cues  supplied)  and  retrieval  collisions  (more  than  one  concept  matches  the  cues); 
more  details  are  given  in  (Touretzky  it  Geva,  1987). 

0.  Discussion 

DUCS  is  a  content-independent  architecture:  it  treats  all  slot  name  and  slot  filler  patterns  identically. 
Although  it  is  convenient  to  assume  that  these  patterns  will  be  micro-feature  based,  so  that  symbols 
with  similar  meanings  (like  TRUNK  and  NOSE)  have  similar  patterns,  at  present  the  model  contains  no 
knowledge  about  individual  microfeatures.  If  it  did,  it  could  use  this  knowledge  to  further  constrain 
the  filler  and  slot  name  patterns,  e.g.,  any  filler  pattern  containing  the  microfeature  human  would  also 
have  to  contain  the  features  mammal  and  animate.  With  an  appropriate  choice  of  microfeatures, 
slot  name  patterns  could  also  be  used  to  constrain  slot  filler  patterns,  e.g.,  the  pattern  for  AGENT 
could  include  a  bit  requiring  that  the  filler  possess  the  animate  microfeature.  This  approach  is  being 
explored  by  Mark  Derthick  in  his  n KLONE  system  (Derthick  1987;  Touretzky  it  Derthick,  1987).  In 
^KLONE  the  microfeature  patterns  are  specified  in  advance  and  all  constraints  between  microfeatures  are 
predetermined  before  the  network  is  built.  It  might  also  be  possible  for  DUCS  or  mKLONE  to  gradually 
learn  (or  at  least  fine  tune)  constraints  between  microfeatures  by  exposure  to  large  numbers  of  examples. 
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The  architecture  of  DUCS  at  the  slot  level  bean  some  resemblance  to  the  distributed  ACAM  (associa¬ 
tive  content  addressable  memory)  recently  described  by  Baum,  Moody,  and  Wilczek  (1987).  The  choice 
of  a  different  random  f?-bit  subset  of  the  slot  name  for  each  column  of  selector  units  serves  the  same 
hashing  function  as  the  prime  factoring  approach  they  propose.  The  ACAM  is  a  hetero-assoeiator,  and 
the  number  of  weights  (cf. ,  units  in  DUCS)  grows  linearly  with  the  size  of  the  input  and  output  patterns 
(N  and  F).  The  capacity  of  the  memory  can  be  increased  to  any  desired  level  by  adding  more  units.  In 
contrast,  the  number  of  weights  in  a  Hopfield  auto-assoeiator  grows  as  [N  +  F)2,  assuming  the  patterns 
consist  of  an  A^-bit  slot  name  concatenated  to  an  F-bit  filler.  Since  there  is  no  intermediate  layer  in  the 
Hopfield  net,  memory  capacity  is  also  limited;  a  Hopfield  net  can  only  store  about  .15 (N  +  F)  patterns 
if  at  least  95%  accuracy  of  recall  is  to  be  maintained. 

DUCS  differs  from  the  ACAM,  and  most  other  neural  networks,  in  that  it  is  a  two-level  architecture. 
At  the  concept  level,  entire  concepts  may  be  stored  and  associatively  retrieved.  But  each  concept  is  a 
collection  of  named  slots,  and  slots  are  themselves  accessed  associatively.  The  two-level  structure  is  one 
of  the  key  features  of  the  DUCS  modeL  As  neural  networks  are  applied  to  increasingly  sophisticated 
knowledge  representation  problems,  it  is  likely  that  more  complex,  multi-level  representations  will  be 
required. 
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